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ABSTRACT: In this paper we present the classical results of Kolmogorov's backward and 
forward equations to the case of a two-parameter Markov process. These equations relates 
the infinitesimal transition matrix of the two-parameter Markov process. However, solving 
these equations is not possible and we require a numerical procedure. In this paper, we 
give an alternative method by use of double Laplace transform of the transition probability 
matrix and of the infinitesimal transition matrix of the process. An illustrative example is 
presented for the method proposed. In this example, we consider a two-parameter warranty 
model, in which a system can be any of these states: working, failure. We calculate the 
transition density matrix of these states and also the cost of the warranty for the proposed 
model. 

Key Words: Markov chain homogeneous with continuous two-dimensional parameter, 
the infinitesimal transition matrix, Kolmogorov equations for homogeneous two-parameter 
Markov chain. 

1 Introduction 

One of the most important elements when we develop the theory about Markov processes 
with continuous time parameter and an enumerable space states, is the characterization 
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that they have with their infinitesimal transition matrix. It is possible to find the matrix of 
transitions probabilities trough the system of differential equations of Kolmogorov. The main 
purpose of this work is to extend the concepts and results to the case of Markov processes 
with continuous two-dimensional parameters (which are usually the time and usage) and 
with an enumerable space states. For this purpose it is necessary to start introducing a 
concept similar to the infinitesimal transitions rates between states, which are defined in the 
case of Markov processes with a continuous-time parameter. In the case of continuous two 
dimensional parameters, the definition of the infinitesimal transition rates is that, in this case, 
they are not the derivatives of the transitions probabilities at time zero. Here it is required 
to work with the second derivatives of the transitions probabilities with respect to each of 
the parameters involved in the two-dimensional point (0,0). Then, with these infinitesimal 
transition rates, new equations are proposed as backward and forward Kolmogorov equations. 

The partial differential equations that are generated, in general, are not easy to solve, 
we use the double Laplace transform to present a simple result. This result however is not 
easy to invert. So, it requires the use of numerical methods for its inversion. After that, we 
present a result to find the distributions of the waiting region and these distributions can be 
used to calculate costs for a previously chosen warranty policy. 

2 Preliminaries 

Definition 1 (MCCTP). A two-parameter stochastic process {X(t,u) \ (t,u) G [0, oo)^} 
with discrete state space S C Z"*", is a Markov Chain with continuous two-dimensional 
parameter (MCCTP), if for all iq, . . . ,ik E S,foT all to < ti < ■ ■ ■ < tk < s < s + t and 
ioT all uq < ui < ■ ■ ■ < Uk < w < w + u, the following equality holds: 

P {X{t + s,u + w) = j \ X{s, w) = i, X{to, uq) = io,..., X(tk, Uk) = ik} = 

P{X{t + s,u + w)=j\X{s,w) = i}, (2.1) 

Definition 2 (MCHCTP). The Markov Chain with continuous two dimensional parameter 
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{X{t,u) I e [0,oo]2} is homogeneous (MCHCTP), if 

Pij(t, u) := P {X{s + t,w + u) = j \ X{s, w) = i} = 

P {X(t, u)=3\ X(0, 0) = z} , V,,^>o (2.2) 

Remark 1. Note that 

< < 1, Vijes,Vt,„>o and ^Pij(t, u) = 1, , V^gs, Vt,„>o (2.3) 

Remark 2. Let t, m > and j G 5", then, 

7r,(t,M) :=P{X(t,M) =j} 

= 5^P{X(t,n)=j, X(0,0) = z} 



= ^ P {X(t, w) = J I X(0, 0) = ^} ■ P {X(0, 0) = 1} 
So, 7r,(t,u) = J2p^At^^) ■ ^i(O'O) (2-4) 



Also, 



^7ij{t,u) = l (2.5) 

i65 



Definition 3 (The transition probabihty matrix). Let t, u > 0, let us definite the transition 
probability matrix by P(t, u) = {pij{t, u) ) and the probability row vector by 7r(t, u) = 
(^7rj{t,u)j . Then, (12. 4p can be written as: 



7r(t,u) = 7r(0,0) ■ P(t,u). (2.6) 
Remark 3. Note that P(0, 0) = I, where I is of identity matrix. 

Definition 4 (Initial probability vector). The vector 7r(0, 0) is called the initial probability 
vector. 
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3 The waiting region for a change of state 

Definition 5 (The waiting region). Suppose that {X(t,u)} is a MCHCTP and that in the 
point (to,uo) = (0,0), the state of the process X{to,uo) = X(0,0) = i, is known. The 
time-use taken for a change of state from state i, is a random vector, say {Ti,^i) which is 
called the waiting region for a change of state from state i. 

Notation 1 (cdf). If i G 5", and t,u > 0, we write the cumulative distribution function of 
the waiting region (rj,7j) as: 

Fi{t,u) ■.= P{n<t,%<u \ X(0,0) =0). 

Also, we write 

Fi{t,u) :=P(r, >t,7, >M I X(0,0) = 0). 
Proposition 1. Suppose that X(t; u) is a MCHCTP. If i G S, and s,t,w,u > 0, then, 

Fi{s + t,w + u) = Fi{t, u) ■ Fi(s, w) (3.1) 
Proof. First of all, let us realize that 

P{T,>s + t,-fi>w + u\ X{0, 0) = i} 

= P {Ti > S + t, Ji > w + u \ X{0, 0) = i,Ti > S, > w} ■ 

P{n>s,ji>w\X{0,0)=i} (3.2) 

Now, since {X{t,u)} is homogeneous, then 

P {Ti > s + t, > w + u \ X(0, 0) = i,Ti > s, 7j > w} 
=P {Ti > s + t, 'ji > w + u \ X{s, w) = i} 

=P {Ti > s + t,^i > w + u, X{s + t,w + u) = i \ X{s, w) = i} (3.3) 
=P {Ti >t,-fi> u, X{t, u) = i\ X(0, 0) = i} 
=P{Ti>t,-fi>u\X{0,0) = i} 
then by (13.31) . the equality (13. 2p can be written as (13. ip . □ 
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Remark 4. In the paper [2] of Marshall and Olkin, we may see that if X and Y are two 
random variables, such that 

P{X > s + t,Y >w + u) = P{X >t,Y >u)- P{X >s,Y >w) (3.4) 

for all s, t,w,u> 0, then: 

P(X > t,F > m) = exp I - Alt- Aaw}, u,t>0 (3.5) 

for some Ai, A2 > 0. 

Proof. In univariate distribution theory is known that if F is a positive random variable, 
then Y ~ exp(A), for some A > 0, is equivalent to 

Fy{s + t) = Fy{s) Frit), for all s,t>0, (3.6) 

where -Fy(s) := P(Y > s). 

Now, let s,t,w,u > and suppose (13.41) . then: 

F(s,0) = P{X > s,Y >0) 

= P{X > s) (3.7) 
= Fxis) 

So, 

Fx{s + t) = F{s + t,0) by(l32D, 
= F(s,0)F(t,0) by (I33D, 

= Fx{s)Fx{t) by(l32D. 

Therefore, X ~ exp(Ai), for some Ai > 0. Similarly, F{0,u) = Fy{u), Fy{w + u) = 
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Fy{w) Fyiu) and Y ~ exp(A2), for some A2 > 0. Finally, 



F{t,u) = F{t + 0,0 + u) 



F{t,0)F{0,u) 



by im, 



Fxit) Fy{u) 



□ 



Proposition 2. Suppose that X(t; u) is a MCHCTP. IfieS, then 



Fi{t, u) = exp <^ - Aii t-\2iU 



t,u>0 



(3.8) 



for some Xu, \2i > 0. 

By using the proposition [T] and the remark HI the result is immediately. 

4 Kolmogorov Equations 

Next it is going to enunciate and demonstrate the important result of Chapman-Kolmogorov, 
for the case of Markov chains with two parameters. 

Theorem 1 (Chapman Kolmogorov equations). // {X(t, m)} is a MCHCTP and let he 
i, j G S, t, s,w,u> 0, then: 




(4.1) 



or in matrix notation 



P(t + s,u + w) = P(t, u) ■ P(s, w) 



(4.2) 
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Proof. We have that, 

Pij{s + t,w + u) = Pr {X{s + t,w + u) = j I X(0, 0) = i} 

= ^ {X{s + t,w + u) = J, X{t, u) = k\ X{0, 0) = i} 

= Pr {X{s + t,w + u) = j \ X{t, u) = k, X(0, 0) =i}- Pr {X{t, u) = k \ X{0, 0) = i} 
keS 

= ^ {X{s + t,w + u) = j \ X{t, u)^k}- Pr {X{t, w) fc I X{0, 0) i} 
keS 

(because {X(t,u)} is a Markov chain) 



= ^ Pr {X{s, w) = j I X{0, 0) = fc} • Pr {X{t, u) = k\ X{0, 0) = i} 

keS 

(because {X(t,u)} is a homogeneous chain) 
So (14. ip is obtained and (14. 2p results then immediatly. □ 

Definition 6 (Infinitesimal transition between states). If we suppose that for t = or 
u = 0, P{t,u) = I, then we define the infinitesimal transition from state i to state j, as 
aij = ^q^q'^ (0;0)- Also, it is defined the infinitesimal transition matrix as the matrix 
A = {o,ij).j^g. 

Remark 5. Note that: 

= lim [Pijih, k) - pij{h, 0) - Pij{0, k) + ^^^(0, 0)1 
h^o+ hk^ ' 

A; -5-0+ 

Q -r • / ■ r Pij{h,k) Pii{h,k) - 1 

bo, n t aij = lim — and an = lim — 

h^o+ hk h^o+ hk 

k^0+ k^0+ 

These relations show that aij > if i 7^ j, and than ajj < 0. Therefore, 

If i 7^ j, Pij{h, k) = hk ttij + o{h) o{k) 
and Pii{h., k) = 1 + hkaij + o{h) o{k) 



Moreover: 



lim 

k^0+ 



Pij{h, k) - pij{h, 0) - pij{0, k) + pij(0, 0) 



hk 



J2p^J{h,k) - J]p,,(/i,0)- J]p,,(0,A;)+ J] p., (0,0) 
jes jes jes 



h^o+ hk 

lim -L [ 1 - 1 - 1 + 1 ] = , by (EJD 

h^o+ hk 



So, 



(4.3) 



Theorem 2 (Backward Kolmogorov equations). If {X{t,u)} is a MCHCTP, i,j & S , t,u > 



0, P{t,u) = \pij{t,u)] is the transition probability matrix and A = ( ctij ) 
infinitesimal transition matrix, then: 



is the 



Lies 



u 



or in matrix form: 



d'^ Pijjt, 
dtdu 



d^P{t,u) 
dt du 



aikPkjit, 



u 



(4.4) 



kes 



A-P{t,u). 



(4.5) 



Proof. Recall the Chapman Kolmogorov equation (14. ip . we have that: 

Pij{t + s,u + w) = ^pik{t,u) ■pkj{s,w) 
kes 

If x(t, s) = t + s and y{u, w) = u + w; and differenting both sides of the before equation with 
respect to t, it is obtained: 

dpij I \ 9x ,^ ^ ^ dpij , dy ^ ^ dp. 



y kes 



So, 



^ (x, y) = ^ ^ (t, n) ■ pkj (s, w) 



kes 



Now differenting both sides of the before equation with respect to u, it is obtained: 

^ (X, ,) • ^ (M) + ^ (X, ^) • ^ -) = E 9^ ^) • p'^^i''^^ 

Therefore, 

Pij \ Pik 



dydx ' dudt 
" kes 



But, taking t = and m = 0, then 



dwd s ' dudt 

kes 



Ojfc ■pkj{s,w) 



k€S 

and thus the theorem has been proved. □ 



Similarly can be proved the next theorem: 

Theorem 3 (Forward Kolmogorov equations). If {X{t,u)} is a MCHCTP, i,j E S, t,u> 
0, P{t,u) = {pij{t,u)) is the transition probability matrix and A = (ajj) is the 
infinitesimal transition matrix, then 



d^Pijjt, 
dt du 



u 



^ Pikit,u) akj, (4.6) 



kes 



or in a matrix form: 



d^P{t,u) 



P(t,n)-A (4.7) 



dtdu 

Notation 2 (Laplace transform.). If g{x), x > 0, is any function for which exists its Laplace 
Transform, then we shall write g*{s) = C{g{x)) = e~*^' g{x) dx if s > for the Laplace 
Transform of g{x). Similarly, if g(x) = {^gij{x)^ is a matrix function for which all its com- 
ponents have Laplace Transform, then we shall write g*(s) = £(g(x)) = {^*j{s)^ for the 
Laplace Transform of g(x). 
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If k{x, y), X > 0, y > is any bivariate function for which exists its bivariate Laplace Trans- 
form, then we shall write /c**(si, S2) = y)) = e~^^^~^^y k{x,y) dx dy if si,S2> 
for the bivariate Laplace Transform of k{x,y). Similarly, if k(x,?/) = {kij{x,y)^ is a ma- 
trix function for which all its components have Laplace Transform, then we shall write 
k**(si,S2) = £^(k(x,y))) = ( S2) ) for the bivariate Laplace Transform of k(x, y). 



Now, we are going to give a relation, between A and P, by using the double Laplace 
transform. 

Theorem 4. // {X{t, u)] is a MCHCTP, and let be Si, S2 > 0, P(t, u) = (pij{t, u)) the 
transition probability matrix and A = ( aj,- ) the infinitesimal transition matrix, then 



P**(S1,S2) = (siS2l- A 



-1 



(4i 



where I is the identity matrix. 

Proof. Let be t, m > 0. Remember that in definition [6] we had supposed that P(t, m) = I if 
t = or M = 0. Now, let us call H(t,M) = ^(t,u). then (14.51) can be written as 

OH 

— it,u)=A-P{t,u) 
and taking Laplace Transform in both sides, with respect to variable t, 



si H*(si, u) - H(0, u) = A- P*(si, u) 



(4.9) 



But, 



dP 1 r 

H(0, u)= — (0, u) = lim - P(0, u + k) - P(0, u) 

OU k^O k I 



lim — ^ 

k-fO k 



I - I 







where is the null matrix. So (14. 9 p can be written as: 



siH*(si,m) = A-P*(si,u) 



(4.10) 
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Now, we observe that: 



poo poo pj 

H*(si,m)=/ e~'''Il{t,u)dt= e~''' —P{t,u)dt 
Jo Jo du 

= — / e-''^Pit,u)dt=—P*isuu) 
du Jq ou 



Then fl4.10p can be written as: 

Si^P*(si,n) = A-P*(si,n) 
ou 

And taking Laplace Transform in both sides with respect to variable u: 



Sl 



S2P**(S1,S2)-P*(S1,0) 



A-P**(Si,S2) 



But, 



P*(si 



poo rc 

0) = / e-'''P{t,0)dt = 
Jo Jo 



e-^'^ldt 



e-'''dt -1 = — - I 

/ Sl 



So ( I4.1ip can be written as: 



S2P**{SI,S2) I 

Sl 



A-P**(S1,S2) 



and solving this equation for P**(si, S2), we obtain (14. Sp . 



(4.11) 



□ 



5 Application 

A high technology machine to produce juice has several identical components. It can work 
if at least one of its components is in a good condition. However, just damaged one of its 
components, it is removed and repaired. When one or more of its components are being 
repaired, the machine does not allow a full work. Once it is fixed, it is placed back into the 
machine. In this sense, the probability that all the components simultaneously are damaged, 
is practically zero. Moreover, not all the time, the machine has the same amount of work. 
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Therefore, the machine has a measurer that records the amount of work done by the machine. 
For this reason, the warranty pohcy takes into account both the time since the machine is 
running, and the amount of work done. For the model that is being analyzed, we suppose 
that X{t, u) is a MCHCTP and that its states space is 5 = {0, 1}. 

It is interpreted as follows: the state 1 means that the machine is working full capacity, 
that is, all its components are in good condition. The state means that at least one of the 
components is being repaired and the machine is not working full capacity. The parameter 
t represents the total time (in years) since the machine was started. The parameter u 
represents the amount of work (in million of juice liters) that the machine has performed. 



The infinitesimal transition matrix of the process {X{t,u)} is 



A 



-2 
0.6 



2 

-0.6 



Therefore, by 



p** I 



SiS2(5SiS2 + 13) 



5siS2 + 3 10 
3 5(siS2 + 2) 



So, by using the method for to invert the double Laplace Transform that was showed in [1] 
by Moorthy, for instance, we have found the next results, which have relative errors less that 
the 4% : 



P(0.2,0.6) 



0.7781 
0.0666 



and 



0.2219 
0.9334 



P(2.0,2.0) 



0.4272 
0.1726 



0.5754 
0.8300 
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In this application, we suppose that the initial probability vector is 7r(0,0) = [O, l] and 
then, we obtain that, 

7r(0.2,0.6) = [0.0666, 0.9334], 

and 

7r(2.0,2.0) = [0.1726, 0.8300]. 

Suppose that the state of the process when t = and n = is X(0,0) = i. Let (rj,7j) 
the waiting region for a change of state from state i and let: 



F{t,u 
G{t,u 
F{t,u 
G{t,u 

f{t,u 



Pr (to < t, 7o < u) 
Pr (ti < t, 7i < u) 
Pr (to > t, 7o > u) 



Pr (ti > t, 7i > u) 



and 



g{t,u) 



8PG_ 
dtdu 



{t,u). 



Then, we can write the next integral equations: 

ru ft 



Poo{t, u) = F{t, u) 
Poi{t,u) = 
Pio{t,u) = 
Pii{t,u) = G{t,u) 



^0 

u ft 



JO 

u ft 



JO 

u ft 



JO 



Pio{t -i,u-uj) dF{^,u) 
Pn{t- ^,u-uj)dF{^,u) 

PQi{t - i,u - uj) dG{i,uj) 



(5.1) 
(5.2) 
(5.3) 
(5.4) 
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By taking double Laplace Transform on (15. 2p and (15. 3p . we obtain: 

Polisu S2) = f**{si, S2)pn{si, S2) 
PToi^u S2) = g**isi, S2) p*o*o{si, S2) 
Since P**(si, S2) is known already, then by (15. 5p and (15. 6p . we deduce that 

S2) = 10/ (5siS2 + 10) and g**{si, S2) = 3/ (5siS2 + 3) 

and so 

F**(si, S2) = 10/ (siS2(5siS2 + 10)) and G**{su S2) = 3/ (siS2(5siS2 + 3)) . 
Again, we use [4] for to invert these double Laplace transforms. 

The warranty conditions are as follows: If damage in one of the machine components occurs, 
within the first six months after putting the machine in use and before it produces 200, 000 
liters of juice, the machine provider changes the item immediately by a new one whose com- 
ponents have been fully checked prior to installation and they meet the quality requirements 
(ie better than new). However, if damage occurs outside the above ranges, but during the 
first year of operation of the machine and before the machine produces 300, 000 liters of juice, 
the machine provider agrees to make the change of the just damage component immediately 
and to make a general revision of the machine to state it better than new. Once one of these 
has been done, the machine provider does not offer more warranty service. Suppose that 
the cost of the machine is C and the cost of the change of one of the components and the 
general revision is 1 = IOC. Then, by using the results proposed by Dimitrov et al [1], the 
expected warranty expense is: 

EWE = C ■ G(0.5, 0.2) + ^ ■ [^(1, 0.3) - ^(0.5, 0.2)] 

C 

= 0.0591(7 + 0.1130— = 0.0704 C 
10 
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(5.5) 
(5.6) 



6 Conclusion 



It is known that the Markov chains with a single continuous parameter are characterized 
by the infinitesimal transition matrix. This study concluded that Markov chains with two 
continuous parameters are characterized by a matrix, that it's also called "infinitesimal 
transition matrix", although its structure is different to the case of a single parameter. In 
addition, this Markov process can be used in a particular type of two-dimensional warranty 
policies problems. 
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